Neural learning for distributions on categorical data

F.X. Albizuri, A.I. Gonzalez, M. Graña, A. d’Anjou University of the Basque Country Informatika Fakultatea, P.K. 649, 20080 Donostia, Spain E-mail: [email protected]; Fax: + 34 943 219306 Abstract. In this paper we define a Boltzmann machine for modelling probability distributions on categorical data, that is, distributions on a set of variables with a finite discrete range. The distribution model is suggested by the log-linear models and it is a generalization of the binary Boltzmann machine. High-order connections are defined instead of hidden units in order to model general probability distributions on multi-valued units. We deduce the iterative learning rule that minimizes the divergence function, which corresponds to a neural scheme. We show that this learning rule converges to the global minimum of the Kullback-Leibler divergence. An example is provided to illustrate the modelling capability of the Boltzmann machine with discrete (non-binary) units.